WebReaper

Introduction

WebReaper is a web crawler which can be used to download objects, web pages or even entire WebSites from the internet unattended. The downloaded objects will be saved in the Windows Internet Cache and can then be browsed using Internet Explorer (v4 or higher) in offline mode, or one of the 'cache browsers' available from various sources.

Alternatively, WebReaper will recreate the entire downloaded website on your local hard disk, even adjusting the links within the site so that it can be viewed directly using the browser of your choice.

  • Features
  • Requirements
  • Getting Started
  • Options and Configuration
  • URL Profiles
  • Feedback/Contact


Features

  • Multithreaded downloading
  • Explorer-style interface
  • User-customisable filters - limit by depth, time per object, total time and 'distance' from starting server, any many others.
  • Simple-to-use filter wizard - helps you build complex filters quickly and easily.
  • Full Drag & Drop support - drag links to/from Internet Explorer/Netscape.
  • Save downloaded files using relative paths to recreate websites stored locally with links adjusted to make them fully browsable.
  • 'Resume' mode reads files saved locally to avoid reloading unchanged pages
  • Proxy & website authentication, allowing websites with passwords or behind firewalls to be reaped.
  • 'URL Profiles' allow depth and filter configurations to be saved with associated URLs for easy re-reaping in future.
  • Command-Line execution to run as a batch process, using a task scheduler (not provided).
  • Full support for ShockWave Flash movies.
  • Works with GetRight ®


Limitations

In order to work WebReaper needs a windows system with an internet link pre-configured. Also, Microsoft Internet Explorer 4 or higher needs to be installed. If you get an error on startup claiming that WinINet.DLL is missing, you need to install/configure IE.


Getting Started

To start downloading the contents of a website, simply click on the Address Bar and type the URL (e.g., http://www.island-of-sark.co.uk).

Click Go and WebReaper will read in the page at that address, extracting a list of all links within that page. Once the page has been completely downloaded, the same process will be performed on each of the 'sub-links' in turn. The structure of the links will be represented in the Tree Pane, with the starting ('root') link at the very top, and sub-links below.

  • Targetting downloads using filters
  • General information about downloads
  • Saving URLs/Filters for use in a later session
  • Resuming downloads from a previous session


The Tree Pane

The tree pane shows the overall hierarchical structure of the pages and objects that WebReaper has found and downloaded. The top item in the tree is the root link, and is the first file found at the starting URL input into the toolbar. As the links on the first page are found, they become sub-items in the tree, and so on, until the download is complete.

Each page is shown in the tree with it's title, or just the URL if the page has not been downloaded. Binary objects (such as graphics, sounds files and so on) are shown with just their URLs. To switch off the titles and see just the URLs, uncheck the Link Titles option under the View menu.

Right-clicking on items in the tree will allow you to open items (either online or - if they've been downloaded - from the locall copy). You can also manually exclude items (and all of their sub-items) from the download.

The icons displayed with each object indicate the type of file; this is dependent on the file associations on your particular machine. So, for example, if you have WinZip installed on your machine, .Zip files will be displayed with the WinZip icon.

The icons also give an indication of the current status of the object. Icons overlaid with a green ticked circle have completed. Those with a red crossed circle failed, and items which are faded have been skipped.

Selecting an object in the tree pane gives more detailed information on that object and its children in the Details Pane.


The Details Pane

The details pane shows the current status information for all sub-links of the item selected in the Tree Pane.

Each item's URL (or title, if available) is shown, along with whatever data is available for it. As the item is downloaded, information such as object size, last modified time, and file type will be displayed. The information column shows the current status of the object, such as whether it has been downloaded, skipped or failed.

The icons displayed with each object indicate the type of file; this is dependent on the file associations on your particular machine. So, for example, if you have WinZip installed on your machine, .Zip files will be displayed with the WinZip icon.

The icons also give an indication of the current status of the object. Icons overlaid with a green ticked circle have completed. Those with a red crossed circle failed, and items which are faded have been skipped.

  • The Tree Pane


Download Notes

General Download Notes

Link Process Order

When downloading, WebReaper follows a set of predefined patterns which govern the order in which files are downloaded. Inline links will always be downloaded first in order to ensure that if the download is aborted or cut short for any reason, the pages which have been downloaded are as complete as possible. Once all inline links on a page (and all other pages at that level in the site links tree) have been completed, only then will the non-inline links be processed. For this reason WebReaper can look quite 'chaotic' as it downloads, but there is method in its madness.


Filters

WebReaper provides various filters to help tailor a site download to particular criteria. Without these filters, WebReaper could (given enough time and disk space) attempt to download the entire World Wide Web!

By using the various components, you can build up complex filters which will target precisely the set of web pages/objects you wish to download.

WebReaper provides components to allow you to filter by:

  • Crawl depth
  • Date last modified
  • Days since last modification
  • Download time
  • File extension
  • HTML/Binary file
  • MIME type
  • File size
  • Servers from root
  • Keywords in URLs
  • Page Contents (keywords)

These filters can be built into a hierarchical tree, producing complex combinations of filters.

  • Constructing Complex Filters
  • Constructing Filters with the Filter Wizard
  • Saving Filters


Filter Types

Crawl Depth

Specifies how 'deep' WebReaper is permitted to crawl for links. If a page contains a link, then the object referred to by that link is a 'child link' of the original page. Each child link represents one level crawl depth. For example, if MYPAGE.HTML has a link to HISPAGE.HTML, which in turn has a link to HERPAGE.HTML, the HERPAGE.HTML is two levels deep from MYPAGE.HTML. The maximum crawl depth specifies the maximum depth to which WebReaper can crawl; when this depth is reached any links within the current page will be ignored.

The exclusions page allows you to specify exact URLs which will not be downloaded during a WebReaper session. Any objects which contain that URL will also be excluded. This feature enables you to exclude entire areas of a site from the reap session.

For example, were you to reap www.otway.com, you could specify www.otway.com/family as a URL exclusion. This would exclude the entire webreaper area from the site (e.g., www.otway.com/family/index.html and www.otway.com/family/familytree.html would both be exluded). You can add URLs by typing them into the edit box and hitting the 'Add' button, or you can drag & drop links from IE, Netscape or the desktop.

Date Last Modified

Filters out files depending on how recently they changed. If most of a website hasn't been updated for several months, but a couple of pages have up-to-date information on them, it would be useful to download only those pages which have changed recently. In particular, this can be used to refresh a site that was reaped a while ago, to update the local copy with recent changes.

Days Since Last Modification

Similar to the 'Date Last Modified' filter, but this is filter uses a relative date rather than an absolute one.

Download Time

This filter sets a maximum download time for each object. Once that time limit has been reached, the download of that object will be aborted.

File Extension

Allows filtering by the file extension. Bear in mind that this is the file extension of the URL of an object (with any extra URL parameters after a '?' or '#' removed), so may not always yield sensible results.

The 'Exclude files with these extensions' checkbox determines whether this is an 'exclusive' or 'inclusive' filter. If it is checked, files with the extensions listed will be excluded from the download. If it is unchecked, only files with extensions matching those listed will be downloaded (i.e., all others will be excluded from the download).

HTML/Binary File

Limits whether HTML files or binary files will be filtered from the download. By adding a filter for both HTML and binary files at the same node in a filter tree (as in the example above) the overall filter behaviour can be subdivided with different filters for binary and HTML files.

MIME Type

The MIME type is the object type (for example "text/html" or "image/jpeg") which is passed back in response to the original request made to the server This filter allows you to limit the download to particular types. Check the box next to each type you want to be included in the download; all unchecked types will be ignored. You can select/unselect groups of MIME types by right-clicking on the list (for example, right-clicking on the "image/jpeg" entry will allow you to check/uncheck all 'image' entries in the list.

If WebReaper should come across any new (previously unseen) MIME types, it will add them to the list of choices for future use. The new types won't be downloaded, but will be available for use in future filters.

File Size

This filter allows the download to be limited to either a maximum or minimum file size. For example, when downloading a site with a lot of graphics, it could filter out large images, so increasing the overall download speed for the site. Alternatively, by specifiying a minimum size

Server Depth

Limits how 'far' away from the server specified in the starting URL WebReaper is allowed to crawl. For example, if the starting URL was, say, www.microsoft.com, then a link from that site to another server (e.g., www.dept-of-justice.com) would be counted as a distance of 'one'. A further link from that site to a third server would be a distance of two, and so on. So specifying a distance of one will allow downloads from all servers referenced on the starting URL's server, but not any others. A distance of two will download from all servers referenced on the starting URL's server, plus any which are referenced on those servers.

Because each page can have a large number of links to other servers, the effect of increasing the server depth option will be exponential. For example, if every page linked to, say, 10 other servers, then a distance of 1 would allow downloads from 11 servers (root + 10 others). A distance of 2 would allow downloads from 111 servers (root + (10 x 10)) servers. Obviously this depends on the site being reaped.

A server depth of zero indicates that WebReaper should restrict downloads to the server on which the starting URL was located.

URL Contents

Similar to the file extension filter, but matches any word/phrase within the entire URL, rather than just the file extension. For example, you may wish to filter out all URLs containing the word 'adserver' which might remove advertising links. Since any part of the URL can be filtered, this can also be used to filter out links on particular servers, or in particular sub-directories.

This filter has a special keyword: "%root%". This keyword will be substituted with the directory of the current starting URL whenever the filter is tested. This allows a filter to be generated to force the download to ignore all files except for those in the same directory as the starting URL.

Keywords

Allows you to filter pages by their actual content. So, for example, you might just want to download links from pages containing the string 'webreaper'.

The string check can be either case sensitive or not, by checking the option on the configuration dialog.


Constructing Complex Filters

When building complex filters, you must realise that the relative positions of the filters is important.

A filter node which is the child of another indicates that an object will be filtered out unless it satisfies both criteria - effectively ANDing the two conditions. For example, in the following filter, only files which are on the root server and which have been modified in the last 2 days will be downloaded.

If more than one filter object is configured at the same level in the filter hierarchy, then these conditions will be OR'd together. That is, only one of the conditions will need to be met for the object to get through the filter. For example, in the following filter objects will be downloaded if either they are binary objectes, or if they have been modified in the last 2 days.

This means that all binary objects will be downloaded regardless of their modification date, but non-binary (i.e., HTML) objects will only be downloaded if they've recently changed. Combining filters like this can allow real flexibility when reaping sites. For example, many sites update their web pages regularly, but may use bitmaps or illustrations which were created a long time ago. In order to reap just the pages which have changed recently, but to still include graphics embedded within those recently changed pages regardless of when the image was created, a filter such as the one above could be used.

Here's an example which shows how a complicated filter can be built up, producing a precisely targetted download:

Firstly, this filter will remove any links whose URL has the word 'adserver' in - saving bandwidth normally wasted by downloading advertisements from automatic servers.

Next the tree branches - one half will only accept HTML files, the other half will only accept binary files. Between the two branches all files will be accepted, but it allows different filters to be configured for each type of file. HTML files are limited to the current server, and will only be downloaded if they have been modified in the last 7 days, whereas binary files can be on the 'next door' server, and must be either GIF or JPEG files to be downloaded.


Filter Wizard

The Filter Wizard

Building filters that target the download precisely as required can be fairly daunting at first. To help set up the most common filters simply and easily, WebReaper comes with an easy-to-use Wizard.

To start the Wizard, select Wizard from the Filter menu, and then follow the on-screen instructions. Once complete, the configuration will be set as your current filter, and can be saved away for future use.


Saving Filters

Once a particular filter configuration is set up, it can be saved in the registry for future use. To do this, choose Save from the Filter menu.

You can manage your stored filters via the Load option on the Filter menu. Filters can either be reloaded, or deleted from the registry

To save your filters permanently into a file on disk (as a backup, or to copy filters to another machine) choose the Export option from the Filter menu. This will prompt you for a text file name, and then will save all currently saved filters to that file.


URL Profiles

A URL profile is a 'template for a download, and consists of:

  • The target URL
  • The filter configuration
  • Selected Save options, including save location

To save a profile, configure the download options and filter, enter the URL, then choose Save Profile As... from the File menu.

If you regularly download a particular website which is updated frequently, a URL profile will enable you to reap the site quickly and efficiently each time.

Double-clicking a URL profile in the Windows Explorer will launch WebReaper and start a download using the settings within that profile.


Options

Download Options

The download options dialog has four pages allowing the general, save, thread and security options to be configured.

Extra commands can be accessed through context menus, by right-clicking on particular items.

WebReaper can also be started with special flags passed into the command-line.

  • General Options
  • Save Options
  • Thread Options
  • GetRight Options
  • Security Options
  • Command-Line Options
  • Context Menu Options
  • Other Options


General Options

Intelligent Dial

If you use a modem to connect to the internet, checking this box will allow WebReaper to automatically dial up a connection when starting a download (if no connection is already open). Some configurations of IE can cause WebReaper to incorrectly asses the state of the connection, and fail to dial out when necessary; if this seems to be happening, you should disable intelligent dial.

User-Agent String

All internet browsers should identify themselves with a 'user agent' string. As well as allowing server administrator to keep statistics on what types of browsers are being used, it can also allow web designers to put browser-oriented content on their pages. So, for example, a page might appear slightly differently when viewed under Netscape and IE.

By adjusting the user-agent string in, you can make WebReaper 'spoof' as another browser, in order to reap the precise content that you require. WebReaper comes with some preset User-agent strings which allow it to mimic popular browsers, including:

  • Microsoft Internet Explorer (the version currently installed on your machine)
  • Netscape Navigator
  • Opera

Log File

WebReaper writes a log of all of its activities to disk, the location of which you can specify here. Leaving the field blank will turn off logging.

Show non-Fatal/Info Log Entries

If this option is checked, extra information will be shown in the logging window, such as which files are currently being downloaded, and which files are being ignored (such as files which are too big/small, too far from the root, etc.). If it is unchecked, only errors or problems will be logged.

  • Download Options


Save Options

WebReaper recreates complete, browsable websites on your local hard disk. The links within the webpages will be adjusted where necessary so that the site will look identical to its internet equivalent when viewed offline with any broweser. However, pages and pictures will load instantly, and no internet connection is necessary to view the site.

Save Copies of Downloaded Files Locally

Toggles file saving on or off.

Location

Specify a fully directory path into which the saved files should be put. Note that if the directory does not already exist, it will be automatically created during the download. To browse for an existing directory, press the "..." button.

Adjust HTML Links to allow local browsing

Checking this option will allow you to create complete working copies of websites on your local hard drive. The links within the pages will be adjusted to 'point' to the local equivalent of their internet target, and picture references will be updated to locate the local copies of any graphics.

As part of this process, a Shortcut will be created in the Save directory specified above which can be used to launch your default browser and start browsing the local site. This shortcut will correspond to the URL at the root of the download.

Resume local files which were partially downloaded

Check this to allow partially downloaded local files to be resumed. If the remote file hasn't changed since the time when the local file was downloaded, the file can be continued from where downloading stopped previously. For example, if a 4Mb zip file was stopped after just 2Mb, WebReaper can resume downloading without having to re-download the first 2Mb. For more information, see Resuming Previous Downloads.

Use Local File If Possible

This option is useful when completing a download which was started earlier. It forces WebReaper to skip the download of any file which exists locally - regardless of whether the remote file still exists or has changed. Local HTML files will be scanned for links and local binaries will be skipped. Bear in mind that files already downloaded will not be refreshed, so this could lead to inconsistencies within the locally saved site.

Save pages to Internet Explorer Cache

Normally, when WebReaper downloads pages it will save them to disk in a directory you specify; the sites won't be 'cached' by Internet Explorer. However, checking this option will cause IE to save the downloaded pages in its own page cache, meaning you can browse the pages in IE directly, as if you'd visited the page earlier in a normal 'online' browsing session.

Obviously, the IE cache takes up disk space, and if you are saving the contents of a website to another location on disk it may be preferable to avoid doing this. Checking this box will disable the caching by WebReaper (IE itself will be unaffected).

Strip HTML tags

Checking this option will cause all HTML tags to be removed from the file as it is saved to disk. This will render the local copy of the site 'unbrowsable', but may be useful for grabbing large chunks of textual information from the web.

Generate Full Paths

If this option is checked, WebReaper will use the path within an object's URL to save the file. For example, if the object is www.webreaper.net\images\screenshot.jpg, and the location directory is set to c:\windows\temp, then the file will be stored in c:\windows\temp\webreaper\images\screenshot.jpg. If this option is left unchecked, all reaped files will be put in the same single save directory. Note that in this situation, any files with the same name will be overwritten by the most-recently processed file.

File Types to Save

This is a list of internet resource types (the same list as on the filters tab). By checking/unchecking individual types, you can control which types of file to save locally, whilst reaping. For example, you may wish to crawl through many websites, saving 'zip' files onto your local machine. This would be useful for a site containing utilities to download. A good way to collect graphics from the web (for use on your web page?) is to select only JPEG files to be saved.

Save All Types

Checking this option is a quick way to force all files to be saved (useful when downloading entire websites). It will also force new (unseen) types to be saved.

  • Download Options


Resume Mode

In order to allow incomplete downloads to be finished at a later date without having to restart from scratch, WebReaper provides resume functionality. These all work when 'Save' mode is turned on in the Save options dialog.

If WebReaper processes a link for which a file already exists locally, it will check the time of the local file against the 'last modified' time of the remote file which is about to be downloaded, before making one of the following choices:

  • If the remote file is older than the local copy of the file, and the file sizes of the local and remote file are identical, the link will be skipped (since the local copy is up-to-date).
  • If the file times match, but the remote file is smaller than the local file, it is assumed that the remote file has been changed, and it will be downloaded from scratch again.

Additionally, if the 'Resume pages which were partially downloaded' option is checked on the Save page of the options dialog, the additional case may happen:

  • If the file times match, but the remote file is larger than the local file, WebReaper will resume the file and continue to read from where it last left off. This allows partially completed filed to be finished after the original download.

Note that if local files are having their links adjusted for local browsing, only binary files will be resumed (since the internal offsets of HTML files may be changed when the links are 'fixed up', causing the file to be resumed from the wrong position).

  • Tips for refreshing sites faster


Resume Notes

Notes on Resuming Downloads

The success of the resume functionality is very dependant on which particular server you're reaping. I've found that some work really well, some don't work at all. It depends on the amount of information the server returns from the initial request. If the server passes back the correct 'last modified' date and the correct file size for a particular object, then resume will work. However, many sites run servers which don't (e.g., for some reason you can get 2k html files being indicated as only having 825 bytes). Of course to ensure files aren't missed, WR is over-cautious and re-downloads anything which could have changed.

There's a couple of ways to speed up resumes, as long as you don't mind if the odd file isn't always refreshed:

  • Turn off the 'resume partially downloaded files' check, any file which already exists locally will be skipped. Obviously, if the file was only partially downloaded originally it will still remain incomplete, but files which have never been downloaded at all will be processed sooner.
  • Increase the number of threads. The HTTP standards don't allow more than 2 connections to a particular server at any one time, meaning that in a lot of cases (particularly when reaping from just one server) increasing the number of threads won't actually increase download performance. However, when resuming, the local files need to be re-scanned; increasing the number of threads will maximise CPU time usage, when scanning these files.
  • Turn off 'adjust html links for local browsing'. If you're reaping a site to gather content rather than the specifically recreate the site locally (e.g., you may be 'collecting' all the bitmaps or Zip files from a particular site) then this will allow more HTML files to be skipped if they've already been downloaded. The reason for this is that when the links are adjusted, the local file is being changed from the remote original file, hence making it impossible to precisely tell if the file needs to be skipped or resumed. For example, if the original link is "/index.html" and the adjusted link is "c:\My Documents\Reaped Sites\www.otway.com\car\index.html" then some 55 bytes have been added to the local file.


Security Options

The security page allows you to enter usernames and passwords for secure websites and any proxy servers or firewalls which might need authentication.

If authentication fails, the option is available to be prompted for a username and password to access the proxy or website.

Internet Authorisation

Takes a username and optional password to use for authentication on secure websites. First enter the username, and then click on the "Set Internet Password" button to enter a password. If a previously entered password exists, the button text will read "Set New Internet Password".

Proxy Authorisation

Takes a username and optional password to use for authentication on proxy servers/firewalls. First enter the username, and then click on the "Set Proxy Password" button to enter a password. If a previously entered password exists, the button text will read "Set New Proxy Password".

Security Note

The passwords will be stored in the registry for future sessions in an encrypted form. However, the encryption is only simple and should not be considered secure. If your password is sensitive, you should either clear the username field before exiting WebReaper (which prevents the password from being stored in the registry) or stick to using the username & password prompt dialog which will be displayed during downloads (and does not store the password).

The author cannot be held responsible for any breach of security caused as a direct or indirect result of using WebReaper.

  • Download Options


Thread Options

The threads page allows you to set the number of download threads that WebReaper will use, and to configure how WebReaper behaves when the download is completed.

Maximum Number of threads

Webreaper uses several threads to download links. This makes it faster, as one thread can continue work whilst the another waits for a response from the internet server from which the file is being downloaded. Depending on your machine, network connection and the site you are downloading from, different numbers of threads can give different performance.

For example, if lots of small files are being downloaded, having many threads running at a time can be faster - many requests will be sent out to the internet, and so it's useful have more work done whilst waiting for responses. If the site from which you are downloading has a few large files, fewer threads may work faster - once the initial request has been made the file can be downloaded using the entire available bandwidth.

Note that IE (and therefore WebReaper) can only have two connections to an HTTP server open at any one time. This is due to the HTTP standards and is implemented to stop browsers/robots overloading the server. So you will find that even if you have more than two threads running, only two may be active at any one time. It is possible (though not recommended) to override this - more details can be found on Microsoft's MSDN site.

Inter-object Delay

In order not to overload servers whilst reaping a site from them, it is courteous to pause in between pages/objects. This stops webreaper from hogging the entire host server's network bandwidth. To do this, set WebReaper to use just one thread, and then specify an inter-object dely of one or two seconds. This field is disabled if more than one thread is to be used.

Maximum Download Time

Allows you to specify a maximum total time for the download to take. The time remaining will be displayed on the status bar during the download.

When Download Completes

    Hang up modem

    To save on telephone bills when leaving WebReaper unattended for longer downloads, you may want to check this option so that when the download completes WebReaper closes the internet connection and hangs up the modem.

    Restore Window

    If WebReaper is downloading in the background, you may want to minimize the window and continue working with other applications. This option forces WebReaper to restore itself to notify you when the download completes.

    Play Sound

    This option allows you to specify a sound (.WAV) file to be played when the download completes. You can browse for a file by clicking the "..." button.

    Shut Down Computer

    If your PC supports Advanced Power Management features, you can select this option so that the machine will shut down, hibernate or go into standby (suspend) mode when the download completes. This is useful when leaving WebReaper performing an unattended download.

Allow Current Files to Finish

If a download is aborted before it completes (by pressing the Stop button on the toolbar), WebReaper will abort any files which are currently downloading and stop as quickly as possible. However, this may result in partially-downloaded files being left locally. By checking this option, you can ensure that WebReaper will complete all files currently being downloaded before the session is aborted.

  • Download Options


GetRight Options

Go to the GetRight website

GetRight ® is an application by Headlight Software which allows resumable downloads of large files from websites and FTP sites. If you have GetRight® installed, WebReaper can work with it to help you download larger files easier by submitting particularly large files from a reaped site to the GetRight® download list.

The files will be downloaded and saved with the other files from the reaped site. WebReaper can therefore be used with GetRight to setup download lists of large file from a particular site, taking advantage of both WebReaper's powerful filtering and link-following features and GetRight's® flexible download options.

Use GetRight to download files larger than...

Turns GetRight support on/off, and sets the size, in Kilobytes, for the threshold above which files will be submitted to GetRight.

Note that any files submitted to GetRight will not be parsed for links - WebReaper will not follow links withing the files, and will not fix up any links contained within the files so that they can be browsed correctly from the local copy of the website.

Use GetRight for binary files only

This will force GetRight to be used only for non-HTML/Shockwave Flash files, meaning that those files will still be downloaded by WebReaper and scanned for new links to follow.

Save as part of local WebSite/in GetRight's default folder

If the first option is selected, files submitted to GetRight will be downloaded to the same place in the local copy of the website that they would have been placed if WebReaper had performed the download itself. This will retain the layout of the local website.

If you choose to save to GetRight's default folder, the current GetRight save location defaults will be used, and you may be prompted for filename information when the file is submitted to GetRight.

Please note that no responsibility is taken for problems caused by the interaction of WebReaper with GetRight. Also, any GetRight support issues should be directed to Headlight Software.

  • Download Options


Other Options

Offline Mode

This option (under the File Menu) will switch to offline mode. This will allow you to crawl web objects currently stored in your local internet cache. This is mainly used for testing.

Clear URL History

this option (under the File Menu) will clear all URLs in the history of the address combo.

Clear Links

if a reap session is stopped before completion, WebReaper will switch to 'resume mode'. The status bar indicates how many links are remaining, and hitting 'Go' again will continue where the previous session left off. If you want to start a new download, simply enter a different URL, or use the Clear Links menu item.


Context Menus

Right-clicking in the WebReaper window will display a context menu, allowing you to perform extra actions or to configure the views.

Clear Log

Selecting this option will clear the contents of the Log Pane.

Include URL

Only available within the tree view during a download, this option allows you to dynamically add and remove branches of links from the download.

For example, if during a download you noticed that a link had been followed which you did not want reaped, by unchecking the 'Include' option on the context menu that link and all its child links would be skipped by WebReaper. The branch can be re-included by checking the menu option again.


Command-Line Options

WebReaper can be started passing a URL as a command-line parameter, which will start the download process immediately. This feature can be used to initiate batch downloads using 3rd-party scheduling software.

The command-line argument '-q' will force the application to quit when the download (or downloads, if batch mode is used) are complete.

The command-line argument '-logurls' will turn on URL logging. During a WebReaper session, every distincy URL that is found will be logged to the file webreaper_url.log in the current directory from where WebReaper is running.

Batch Mode

WebReaper can be set to run in 'batch mode', where it will process a list of starting URLs, performing a full 'reap' on each URL in turn.

To do this, you must create a text file containing the list of URLs, one on each line. For example, list.txt:

    www.otway.com
    http://www.microsoft.com
    www.codeguru.com

If you now run "webreaper.exe list.txt", the URLs in the file will be processed in batch mode. Note that a batch file can also be made up of fully-specified filenames of URL profiles, making it possible to use a different filter configuration and save configuration for each item in the batch list.


Frequently Asked Questions

Q: Do I need to have Internet Explorer installed to use WebReaper?
Yes. WebReaper uses Microsoft's WinINet libraries which are only available as part of Internet Explorer. WebReaper also uses the IE proxy and dial-up configurations to access the internet. You can download the latest version of IE here.
Q: Do I have to use IE to view the locally saved websites?
No. Although WebReaper requires IE in order to access the internet, you can continue to use any browser for surfing. The pages stored locally are valid web pages, and can therefore be viewed with any browser.
Q: Some sites result in an "Access denied" error in WebReaper. How can I download them?
Unfortunately, you can't. WebReaper obeys the internet Robots Exclusion Standard, which was set up to give website owners some control over which web-crawlers and spiders can access particular areas of their sites. This is controlled by a file (robots.txt) which will be in the root directory of the website.

Some site owners ban robots to stop people downloading the material on their sites en masse. Others do it because they have limited bandwidth available, and robots can thrash their servers (causing bandwidth problems with their sites).
Q: Can WebReaper download ASP and CGI pages?
Yes, in most cases. WebReaper downloads pages in precisely the same way that IE does, so if IE can download a page then WebReaper will usually download it too. However, some sites may be structured in such a way that the saved pages may not match what you see when online.
Q: Does WebReaper support cookies?
Yes - Any cookies generated within an Internet Explorer session will be used by WebReaper. So if you have local cookies which allow you to access a protected site without having to enter user information, WebReaper will also be able to use them.
Q: Does WebReaper handle ShockWave Flash movies?
Yes. Movies will be scanned, and internal links followed. The links will also be adjusted so that the locally-saved website can be viewed correctly, including movie content.
Q: Does WebReaper handle JavaScript?
No, not yet. Parsing JavaScript and downloading all pages that are referenced within the code is very complicated. However, I plan to add some JavaScript support soon.
Q: Is WebReaper free?
Yes. You can use WebReaper without paying to register for as long as you like, as long as you aren't using it in a commercial environment. No features are disabled, and you will get full support with any queries or suggestions that you might have.
Q: Why don't you force users to pay for WebReaper?
I originally wrote WebReaper as a utility for myself, because I could find no similar app on the Web which was available for free. I decided to make the application available to others as I thought it would be useful.
Q: Why should I register?
I work as a commercial developer and consultant, and have created WebReaper in my spare time as a hobby. Unfortunately, such is the popularity of WebReaper that I am answering 10 or so support emails per day. This takes a lot of time - time which I would normally spend earning! So, in order to help WebReaper cover its costs, I ask that you register your copy if you use it regularly.

The registration fees also help pay for the site-hosting costs.
Q: If I register, will any of my details be given out to anyone else?
No. The registration/credit card authorisation is handled by JRiver, a third-party company who provide secure on-line payment. The registration details are sent to me, and I will not divulge them to any third parties.
Q: Why is WebReaper no longer adware?
Unfortunately, as a result of the 'dotcom collapse', web-based advertising is no longer generating decent revenue. In particular, the amount of support mails generated by having the advertising in the app was far outweighing any financial benefits it was creating. So I decided to drop the ads and return WebReaper to real freeware status, hoping that regular users will show their appreciation of this by registering.
Q: I registered WebReaper to get rid of the ads. Now it's freeware again, can I have my money back?
Registration pays for a licence to use the application - it is not simply payment for removing the adverts. The $40 you paid has bought you a lifetime licence to use WebReaper.
  • Click here for the WebReaper Online FAQ


The Author

This software was written by Mark Otway.

Copyright © 1998-2000 Bluebeam Ltd.

Feedback

If you have any questions, ideas, or problems with WebReaper, please mail them to info@webreaper.net.

Update Announcements Mailing List

WebReaper will automatically check the home page periodically for new releases, notifying you immediately of any new versions as they become available.

For up-to-the-minute news and announcements about forthcoming releases, you can subscribe to the WebReaper mailing list.

You can join the list at the eGroups WebReaper site.

Alternatively, you can subscribe by sending an e-mail to webreaper-subscribe@egroups.com.


Registration

WebReaper is Freeware for non-commercial use. You may use the fully functional application for as long as you like, without restriction, and with support from myself.

However, if you like WebReaper, find it useful, or use it in a commercial environment, please register your copy, giving you peace-of-mind, safe in the knowledge that you've contributed to the development of the application.

Registration costs just $40, and can be done from within the application (within the About dialog box, under the Help menu), using a secure credit-card payment system.


Credits

Quite a few people have contributed to the success of WebReaper. Here's where I thank them...

  • Kirk Stowell of CodeJock.com who wrote the CJLibrary, without which the wonderfully flexible dockable toolbar system would have been impossible.
  • Jordan Russell, whose excellent Inno Setup made writing an installer for WebReaper a breeze.
  • J. River Inc., who provide the secure credit-card transaction functionality to all users to register WebReaper.
  • Lee at Simply Web Design, whose assistance whilst redesigning the website was invaluable.
  • Iain at Iain Dean Design, who also helped with the website design.
  • Markus Oberhumer & Laszlo Molnar for their clever executable compressor, UPX, which shrunk the WebReaper executable by over 60%, saving around 0.5Mb of disk space once installed.

And, of course, thanks to all the users who have provided feedback and ideas to keep WebReaper ahead of the competition....


Licence

This software is released as freeware for non-commercial use, but it may not be sold or be included in a product or package which intends to receive benefits through the inclusion of this software. This software may be distributed freely over the Internet as long as the file(s) remain unmodified. This software may not be included on CD-ROMs without prior permission from the author.

Disclaimer

This software and the accompanying files are distributed "as is" and without any warranties whether expressed or implied. no reponsibilities for possible damages or even functionality can be taken. the user must assume the entire risk of using this program.